fix: improve error messages for missing database tables by ivanauth · Pull Request #2775 · authzed/spicedb

ivanauth · 2025-12-16T01:04:10Z

When users start SpiceDB without running migrations first, they get a confusing error like:

unable to list namespaces: ERROR: relation "namespace_config" does not exist (SQLSTATE 42P01)

This doesn't tell them what went wrong or how to fix it.

Now the error message is clear and actionable:

datastore error: the database schema has not been initialized; please run "spicedb datastore migrate": ERROR: relation "namespace_config" does not exist (SQLSTATE 42P01)

This applies to all SQL datastores (postgres, mysql, crdb) and covers read operations, write operations, and statistics queries.

Closes #2749

codecov · 2025-12-16T01:07:39Z

Codecov Report

❌ Patch coverage is 82.84024% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.66%. Comparing base (4c9d349) to head (f918ab5).
⚠️ Report is 53 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/datastore/spanner/errors.go	0.00%	18 Missing ⚠️
pkg/middleware/readiness/readiness.go	89.37%	8 Missing and 2 partials ⚠️
pkg/cmd/server/server.go	0.00%	1 Missing ⚠️

❌ Your project check has failed because the head coverage (73.66%) is below the target coverage (75.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2775      +/-   ##
==========================================
- Coverage   74.90%   73.66%   -1.24%     
==========================================
  Files         484      493       +9     
  Lines       57938    60552    +2614     
==========================================
+ Hits        43394    44597    +1203     
- Misses      11519    12798    +1279     
- Partials     3025     3157     +132

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

miparnisari

Used Postman and tried all the APIs against CRDB and Spanner. Notes:

we're missing this change for Spanner. You can use docker-compose -f docker-compose.spanner.yaml up --build to test. E.g.
- call to WriteSchema I get "unable to list caveats: spanner: code = \"NotFound\", desc = \"Table not found: caveat\", requestID = \"1.23ff9b5081a896f8.1.1.70.1\"",.
- call to DeleteRelationships I get "object definition doc not found". This line has to be updated to account for when the error description says "table"
  
  spicedb/internal/datastore/spanner/reader.go
  
  Line 300 in f41f641
  
  if spanner.ErrCode(err) == codes.NotFound {

in many cases, the errors returned to the user are prefixed with error encoding zedtoken: unable to read unique ID which makes no sense to me, and probably not to the end user 😆 It comes from

spicedb/pkg/zedtoken/zedtoken.go

Lines 71 to 74 in 01d72a1

    
           datastoreUniqueID, err := ds.UniqueID(ctx) 
        
           if err != nil { 
        
           	return nil, fmt.Errorf(errEncodeError, err) 
        
           }

. i'm very tempted to remove that error encoding zedtoken and to clarify what the "unique ID" is

in spanner, if you didn't run migrations, calling Watch api gives a HTTP 400. But against CRDB, the watch service is completely disabled: unknown service authzed.api.v1.WatchService, which means that you need to re-boot the server after you ran the migrations. I wonder if the server is supposed to be hable to handle migrations running at the same time ?

Anyway. Could you update

spicedb/internal/datastore/crdb/crdb.go

Line 576 in 3370198

    
           features.Watch.Reason = "Range feeds must be enabled in CockroachDB and the user must have permission to create them in order to enable the Watch API: " + err.Error()

to set a different error message in this case?

when you call ImportBulk, you get this

{
    "code": 2,
    "message": "statement description failed: ERROR: relation \"relation_tuple\" does not exist (SQLSTATE 42P01)",
    "details": []
}

, seems like we missed an error handling here:

spicedb/internal/datastore/postgres/common/bulk.go

Line 79 in 3370198

    
           copied, err := tx.CopyFrom(ctx, pgx.Identifier{tupleTableName}, colNames, adapter)

miparnisari · 2025-12-23T17:44:30Z

internal/datastore/common/errors.go

+		err,
+		codes.FailedPrecondition,
+		spiceerrors.ForReason(
+			v1.ErrorReason_ERROR_REASON_UNSPECIFIED,


nit: please create the right error code

miparnisari · 2025-12-23T17:45:07Z

internal/datastore/common/errors.go

+// instructing the user to run migrations.
+func NewSchemaNotInitializedError(underlying error) error {
+	return SchemaNotInitializedError{
+		fmt.Errorf("datastore error: the database schema has not been initialized; please run \"spicedb datastore migrate\": %w", underlying),


Suggested change

fmt.Errorf("datastore error: the database schema has not been initialized; please run \"spicedb datastore migrate\": %w", underlying),

fmt.Errorf("%w. please run \"spicedb datastore migrate\", underlying),

miparnisari · 2025-12-23T17:51:18Z

internal/datastore/crdb/caveat.go

+		if wrappedErr := pgxcommon.WrapMissingTableError(err); wrappedErr != nil {
+			return nil, datastore.NoRevision, wrappedErr


Suggested change

if wrappedErr := pgxcommon.WrapMissingTableError(err); wrappedErr != nil {

return nil, datastore.NoRevision, wrappedErr

if pgxcommon.IsMissingTableError(err) {

err = common.NewSchemaNotInitializedError(err)

}

to keep the same pattern as the lines before

miparnisari · 2025-12-23T18:09:09Z

internal/datastore/postgres/common/errors.go

 	pgReadOnlyTransaction       = "25006"
 	pgQueryCanceled             = "57014"
 	pgInvalidArgument           = "22023"
+	pgMissingTable              = "42P01"


there are two constants with the same value defined elsewhere: postgresMissingTableErrorCode

please remove them in favor of this one

miparnisari · 2025-12-23T18:52:57Z

internal/datastore/postgres/revisions.go

 	cterr := tx.QueryRow(ctx, sql, args...).Scan(&newXID, &newSnapshot, &timestamp)
 	if cterr != nil {
+		if wrappedErr := common.WrapMissingTableError(cterr); wrappedErr != nil {
+			return newXID, newSnapshot, timestamp, wrappedErr


Suggested change

return newXID, newSnapshot, timestamp, wrappedErr

cterr = ....

miparnisari · 2025-12-23T18:53:43Z

internal/datastore/postgres/reader.go

 	})
 	if err != nil {
+		if wrappedErr := pgxcommon.WrapMissingTableError(err); wrappedErr != nil {
+			return nil, wrappedErr


Suggested change

return nil, wrappedErr

err = wrappedErr

otherwise you lose the errUnableToListNamespaces below

miparnisari · 2025-12-23T20:21:37Z

I also wonder if instead of littering all the datastore implementations with error handling, we could just prevent the server from coming up online when you have never ran migrate.

E.g. here

spicedb/internal/services/health/health.go

Lines 112 to 115 in 70cd757

    
           if !dsReady.IsReady { 
        
           	log.Ctx(ctx).Warn().Bool("datastoreReady", false).Msgf("datastore failed readiness checks: %s", dsReady.Message) 
        
           	return false 
        
           }

if dsReady.message includes datastore is not migrated: currently at revision \"\", then we could tell people to run migrate. we'd need to write a generic test ensuring that all datastores return empty revision when they haven't been migrated.

I'm sure there are more changes required than this, perhaps an update to Server.Complete to include this check, but @josephschorr does this sound reasonable?

- Add TODO comment for future ERROR_REASON_DATASTORE_NOT_MIGRATED - Simplify error message format to: "%w. please run migrate" - Use IsMissingTableError + reassignment pattern (not early return) - Export PgMissingTable constant, remove duplicates from migration drivers - Add Spanner IsMissingTableError support in reader.go and caveat.go - Differentiate CRDB watch service error message for missing tables - Add ImportBulk error handling in bulk.go - Update tests for new error message format 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add TODO comment for future ERROR_REASON_DATASTORE_NOT_MIGRATED - Simplify error message format to: "%w. please run migrate" - Use IsMissingTableError + reassignment pattern (not early return) - Export PgMissingTable constant, remove duplicates from migration drivers - Add Spanner IsMissingTableError support in reader.go and caveat.go - Differentiate CRDB watch service error message for missing tables - Add ImportBulk error handling in bulk.go - Update tests for new error message format

ivanauth · 2026-01-06T17:31:41Z

I replaced scattered IsMissingTableError checks with a single gRPC readiness middleware that blocks ALL requests until the datastore is migrated.

Before: 15 files with scattered IsMissingTableError checks (easy to miss spots)
After: One middleware gates ALL requests until migrated

- Add TODO comment for future ERROR_REASON_DATASTORE_NOT_MIGRATED - Simplify error message format to: "%w. please run migrate" - Use IsMissingTableError + reassignment pattern (not early return) - Export PgMissingTable constant, remove duplicates from migration drivers - Add Spanner IsMissingTableError support in reader.go and caveat.go - Differentiate CRDB watch service error message for missing tables - Add ImportBulk error handling in bulk.go - Update tests for new error message format

When datastore operations fail due to missing database tables (typically because migrations haven't been run), users now see a clear error message: 'please ensure you have run spicedb datastore migrate' This improves the user experience by providing actionable guidance instead of raw database errors like 'relation X does not exist'. The improvement covers all reader operations across CRDB, PostgreSQL, and MySQL datastores including namespace, caveat, counter, and statistics queries.

- Add TODO comment for future ERROR_REASON_DATASTORE_NOT_MIGRATED - Simplify error message format to: "%w. please run migrate" - Use IsMissingTableError + reassignment pattern (not early return) - Export PgMissingTable constant, remove duplicates from migration drivers - Add Spanner IsMissingTableError support in reader.go and caveat.go - Differentiate CRDB watch service error message for missing tables - Add ImportBulk error handling in bulk.go - Update tests for new error message format

Replaces scattered IsMissingTableError checks (15 files, 157 lines) with a single gRPC readiness middleware that blocks ALL requests until the datastore is migrated. Key improvements: - Single point of control vs copy-pasted checks everywhere - Impossible to miss code paths - all gRPC requests gated automatically - Clear error message: "Please run 'spicedb datastore migrate'" - Cached checks (500ms) with singleflight to prevent thundering herd - Health probes bypass the gate for Kubernetes compatibility Net: -81 lines, better coverage, consistent UX.

Sort imports alphabetically to satisfy gci linter - logmw before readiness. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

ivanauth requested a review from a team as a code owner December 16, 2025 01:04

github-actions bot added area/datastore Affects the storage system area/tooling Affects the dev or user toolchain (e.g. tests, ci, build tools) labels Dec 16, 2025

ivanauth force-pushed the fix/issue-2749-complete-coverage branch from 8ec386b to 126c8ff Compare December 19, 2025 22:47

ivanauth changed the title ~~Improve error messages for missing database tables~~ fix: improve error messages for missing database tables Dec 19, 2025

miparnisari requested changes Dec 23, 2025

View reviewed changes

ivanauth force-pushed the fix/issue-2749-complete-coverage branch from ee34e7f to 7c53e96 Compare January 5, 2026 17:59

ivanauth mentioned this pull request Jan 5, 2026

feat: add ERROR_REASON_DATASTORE_NOT_MIGRATED error code authzed/api#159

Merged

github-actions bot added the area/cli Affects the command line label Jan 7, 2026

ivanauth force-pushed the fix/issue-2749-complete-coverage branch 2 times, most recently from ab8dd24 to 33b2e94 Compare January 7, 2026 15:38

github-actions bot added area/schema Affects the Schema Language area/dependencies Affects dependencies labels Jan 7, 2026

ivanauth force-pushed the fix/issue-2749-complete-coverage branch from 3d37e2d to d873dd4 Compare January 7, 2026 20:39

github-actions bot removed area/schema Affects the Schema Language area/dependencies Affects dependencies labels Jan 7, 2026

ivanauth force-pushed the fix/issue-2749-complete-coverage branch 2 times, most recently from 131f022 to 7176e82 Compare January 23, 2026 21:15

ivanauth force-pushed the fix/issue-2749-complete-coverage branch 2 times, most recently from a03d096 to 10f604e Compare January 24, 2026 00:36

ivanauth force-pushed the fix/issue-2749-complete-coverage branch from 10f604e to c3ce28a Compare January 24, 2026 00:38

ivanauth added 2 commits February 5, 2026 10:35

ivanauth and others added 4 commits February 5, 2026 10:35

chore: update TODO to reference authzed/api#159

39804a8

fix: correct import order in defaults.go

24d2ead

Sort imports alphabetically to satisfy gci linter - logmw before readiness. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

fix: remove private fields from generated ToOption method

2eefa26

tstirrat15 force-pushed the fix/issue-2749-complete-coverage branch from acd0018 to 2eefa26 Compare February 5, 2026 17:35

retrigger CI

f918ab5

ivanauth added the Skip-Changelog label Mar 5, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: improve error messages for missing database tables#2775

fix: improve error messages for missing database tables#2775
ivanauth wants to merge 7 commits intoauthzed:mainfrom
ivanauth:fix/issue-2749-complete-coverage

ivanauth commented Dec 16, 2025 •

edited

Loading

Uh oh!

codecov bot commented Dec 16, 2025 •

edited

Loading

Uh oh!

miparnisari left a comment

Uh oh!

miparnisari Dec 23, 2025

Uh oh!

miparnisari Dec 23, 2025

Uh oh!

miparnisari Dec 23, 2025

Uh oh!

miparnisari Dec 23, 2025 •

edited

Loading

Uh oh!

miparnisari Dec 23, 2025

Uh oh!

miparnisari Dec 23, 2025

Uh oh!

miparnisari commented Dec 23, 2025 •

edited

Loading

Uh oh!

ivanauth commented Jan 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	datastoreUniqueID, err := ds.UniqueID(ctx)
	if err != nil {
	return nil, fmt.Errorf(errEncodeError, err)
	}

	fmt.Errorf("datastore error: the database schema has not been initialized; please run \"spicedb datastore migrate\": %w", underlying),
	fmt.Errorf("%w. please run \"spicedb datastore migrate\", underlying),

		if wrappedErr := pgxcommon.WrapMissingTableError(err); wrappedErr != nil {
		return nil, datastore.NoRevision, wrappedErr

	return newXID, newSnapshot, timestamp, wrappedErr
	cterr = ....

Conversation

ivanauth commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

miparnisari left a comment

Choose a reason for hiding this comment

Uh oh!

miparnisari Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

miparnisari Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

miparnisari Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

miparnisari Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miparnisari Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

miparnisari Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

miparnisari commented Dec 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ivanauth commented Jan 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ivanauth commented Dec 16, 2025 •

edited

Loading

codecov bot commented Dec 16, 2025 •

edited

Loading

miparnisari Dec 23, 2025 •

edited

Loading

miparnisari commented Dec 23, 2025 •

edited

Loading